I wonder if namespaces would be a better name for this now. The other thing that's a little tricky is we're really working with two distinct types of namespaces: score / count data supplied by the investigator and different namespaces for variant data we cache in our database. While we fetch score / count columns via one method, we will fetch data for other namespaces by a different one.

I prefer the namespaces in here. It's clearer. I also considered whether put all of them in one function cause it's quite complex and messy. Separating them makes it easy to distinguish whether the data is from ours or others. We can modify this part when we're ready to add the clinVar and gnomAD's data.

First, we might consider maintaining a dictionary of all of our namespaces. Each namespace keys a list of column names. The idea here is that if we combine the outer key and any column name from the contained list, we get a namespaced column. This will let us both easily generate a namespaced column name while also retaining efficient and easy access to the real internal column name. The other benefit is that we avoid key collisions right from the start of the CSV generation process, so we won't have to worry about identifier columns colliding between say, ClinVar and gnomAD namespaces.

In this suggestion, I also eliminated the dictionaries for the internal mappings of the column keys. I think that we are only ever going to have two of those, and doing each of them explicitly is more readable.

Note that on the last line, I think we want to add the score column whenever scores is in the data types list and we aren't including custom columns. Right now if you supplied [clinvar, scores] and include_custom_columns = False, you wouldn't get the score column as output which isn't what we want.

@EstelleDa One thing @sallybg suggested to me today was to put the hgvs strings into their own, more descriptive, namespace rather than mavedb. I'm tempted by something like hgvs but that might be confusing considering we have other columns that start with hgvs. So maybe mapped_hgvs is most precise. We can put other identifiers into their own namespaces, so I think having an hgvs specific one is best.

The biggest difference in this implementation is here, where we now stop falling back on custom keys in the else block and instead handle all keys in each namespace explicitly. If a column name is in a namespaced columns list and it isn't found by this function, it gets a value of None. I like this though because we are handling exports at the namespace level and we have to explicitly define the export behavior for each key.

This is also where we rename the columns so we can get their namespaces. Since we still have access to the raw column name, it's more readable to do that renaming and we don't need to do anything with string manipulation.

(i renamed the dtype parameter to namespaces.

The nice thing about this implementation is that by the time we get to this point we're already done. We've fetched and renamed all our export files. We just need to generate a list of columns for the drop_na_columns_from_csv_file_rows function.

    rows_columns = [f"{namespace}.{col}" for namespace, cols in namespaced_score_set_columns.items() for col in cols]

One thing that your solution did that this one doesn't do is remove the prefix from columns if only one namespace is being exported. For consistency with the new changes, I think I actually prefer to always export columns inside their namespace. This implementation also would require a few small changes to some of the other CSV export routers, but I think that would be trivial.